Various image processing techniques, such as super-resolution processing for improving the resolution of captured images, haze removal processing for improving the visibility by removing hazes included in the images, and so on, have been introduced into the recent cameras. In particular, Patent Document 1 and Patent Document 2 disclose image processing techniques relating the present invention. These Patent Documents are common in that an image is reconstructed by filtering (averaging) an input image in a time domain. Patent Document 1 is to obtain and correct a deviation (motion vector) of the input image by referring to the reconstructed image (motion vectors) and Patent Document 2 is to output the reconstructed image as it is output.
Here, the technique of Patent Document 1 (Prior art 1) will be described in brief with reference to FIGS. 16 to 18.
In the technique of Patent Document, image processing is performed in the sequence shown in FIG. 16. Specifically, a reference image is first estimated (S51). Next, an optical flow (motion vector) is calculated (S52). Finally, motion correction is performed (S53).
The estimation of the reference image in S51 is performed by exponential moving average of an input image. Since the exponential moving average has the effect of reducing high frequency components of time-series input images, the estimated reference image becomes an image having no motion as a moving body, fluctuation due to shimmer and so on are removed from the reference image,
Next, in the calculation of the optical flow in S52, as shown in FIG. 17, a motion vector of a local area (for example, 5 vertical pixels and 5 horizontal pixels) of the input image is estimated by block matching (BM) between the input image and the reference image. This motion vector is taken as a motion vector of the center point (pixel of interest) of the local area. In the example of FIG. 17, for example, the direction and amount of movement (from a position of the corresponding reference image) of a local area 1 (5 vertical pixels and 5 horizontal pixels) of an input image 172 similar to a local area of a reference image 171 (i.e., taking BM) are represented as a motion vector 1 of the center point (pixel 1 of interest) of the local area 1.
In the motion correction in S53, as shown in FIG. 18, local areas 1 to 3 of the input image are shifted according to the estimated motion vector and are overlapped each other. In this way, a corrected image is generated by the overlap of partial images.